Skip to content

[SM90] Sparse decode: honor topk_length for DeepSeek-V3.2 - #27

Open
LoongPei wants to merge 1 commit into
vllm-project:mainfrom
LoongPei:sm90-v32-topk-length
Open

LoongPei wants to merge 1 commit into
vllm-project:mainfrom
LoongPei:sm90-v32-topk-length

Conversation

@LoongPei

@LoongPei LoongPei commented Sep 28, 2026 •

Copy link
Copy Markdown

Summary

The SM90 sparse decode kernel honors topk_length only for the DeepSeek-V4 layout; for V3.2 it asserts topk_length == nullptr. SM100 already honors it for V3.2. This makes SM90 consistent:

  • The per-request block count comes from topk_length, rounded up to 64 with at least one block, which matches what get_decoding_sched_meta plans.
  • Index positions at or past topk_length are treated as invalid, as in the V4 path.
  • The checks are compiled in only for two new instantiations, v32_persistent_h{64,128}_topklen.cu, selected when topk_length is given (the same scheme as the SM90 sparse prefill kernel). The existing V3.2 kernels are unchanged.

The same kernel change is proposed upstream in deepseek-ai#216 (open, not reviewed yet), also as a compile-time specialization. This PR is for the fork, whose source layout differs, because vLLM needs it for DCP (vllm-project/vllm#58980). If deepseek-ai#216 lands upstream first, a later sync can replace this.

Motivation

vLLM runs V3.2 with decode context parallelism, where each rank owns about 1/dcp of every 2048-wide top-k row. Today the kernel runs on the full row with the other ranks' slots masked as -1. Compacting the rank's slots and passing their count as topk_length (one batch entry per token) skips the masked slots.

Kernel time on one DCP4 rank (H20, h_q=64, top-k 2048, ~512 owned slots per row):

rows masked, 2048 wide (us) topk_length (us) speedup topk_length incl. planning (us)
8 37.0 20.2 1.83x 35.8
16 63.4 26.0 2.44x 42.3
32 110.5 42.6 2.59x 60.4
64 174.9 54.4 3.22x 74.9
128 349.0 108.7 3.21x 132.5
1024 2454.3 687.1 3.57x 754.8

The lengths differ per layer, so vLLM plans every call; the last column includes that planner run. A companion PR, opened at the same time, makes the planner 2.1-3.6x faster at these batch sizes.

End to end in vLLM (8x H20, TP8 + DCP4, vllm-project/vllm#58980), the prefill time of fresh prompts drops by 24-28%, and the decode step time by 3.9-8.3% at 16-32 concurrent requests. At 8 requests the per-layer planning cancels the gain out until the planner is faster.

Tests

  • tests/test_flash_mla_sparse_decoding.py now also generates V3.2 cases with topk_length: 180 new correctness cases. The harness poisons every KV row not selected within topk_length with NaN, so any read past the length fails.
  • All 4896 SM90 correctness cases pass on H20 (420 V3.2, 4476 V4, 2232 with topk_length).
  • No change for callers that don't pass topk_length. FlashMLA's SM90 decode perf cases before/after, best of two runs (us):
case before after
V3.2 b=2 h=128 topk=2048 34.94 34.95
V3.2 b=64 603.92 605.58
V3.2 b=74 690.74 690.86
V3.2 b=128 1193.56 1197.68
V3.2 b=148 h=64 topk=16384 5164.69 5162.33
V3.2 b=148 h=128 topk=16384 10552.32 10546.20

V4 cases are within ±0.9% as well.

Not tested on SM100; this PR doesn't touch SM100 code.

AI assistance (Claude) was used for this change. I reviewed all of it and ran the tests above.

The SM90 sparse decode kernel bounded index reads by topk_length only for
the DeepSeek-V4 layout and asserted topk_length == nullptr for V3.2. SM100
already honors it for V3.2.

With topk_length, the V3.2 kernel now takes each request's block count
from it (matching get_decoding_sched_meta) and treats index positions at
or past it as invalid. The checks are compiled into two new
v32_persistent_h{64,128}_topklen instantiations only, so the existing V3.2
kernels are unchanged.

The sparse decoding test now also generates V3.2 cases with topk_length.

Assisted-by: Claude
Signed-off-by: LoongPei <3136347099@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant